LSAC RESEARCH REPORT SERIES Robust Automated Test Assembly for Testlet-Based Tests: An Illustration With the Analytical Reasoning Section of the LSAT

The Law School Admission Council (LSAC) is a nonprofit corporation whose members are more than 200 law schools in the Council was founded in 1947 to facilitate the law school admission process. The Council has grown to provide numerous products and services to law schools and to more than 85,000 law school applicants each year. All law schools approved by the American Bar Association (ABA) are LSAC members. Canadian law schools recognized by a provincial or territorial law society or government agency are also members. Accredited law schools outside of the United States and Canada are eligible for membership at the discretion of the LSAC Board of Trustees. This study is published and distributed by LSAC. The opinions and conclusions contained in this report are those of the author(s) and do not necessarily reflect the position or policy of LSAC. Executive Summary In many high-stakes tests, subsets of questions (i.e., items) grouped around a common stimulus are often utilized to increase testing efficiency. These subsets of items are commonly called testlets. Since responses to items belonging to the same testlet not only depend on the test taker's ability, but also on the correct reading, understanding, and interpretation of the stimulus, the assumption that the responses to these items are independent of one another does not always hold. A mathematical model called item response theory is often applied in automated test assembly (ATA) with testlets. Testlet response theory (TRT) models have been developed to deal with dependency among items within a testlet. This report addresses some of the questions that arise in the application of TRT models to ATA. Specifically, a robust ATA method is applied. The results obtained by this method, as well as the advantages it offers, are discussed. Finally, recommendations about the use of the new method are given.